Evaluation of “safety-domino”: a Graphical Metaphor for Supporting Minimal Cut Set Analysis

Embedded systems, which are safety critical, are frequently analyzed to find out whether they follow the safety standards and to improve their safety. Fault tree analyses enable the safety analysts to find possible causes of a system’s failure or unsafe behavior. From a fault tree, minimal cut sets are computed giving unique combinations of basic events that cause such a failure. Using minimal cut sets, the safety analysts try to find out how to improve the safety of the system with a restricted amount of resources, e.g., cost and time. The two most important results of minimal cut set analyses are the order (number of basic events) and the failure probability of the minimal cut sets. Small order minimal cut sets having 1-3 basic events are the most important ones, because, e.g., a single point of failure is more likely to occur than multiple failures at the same time. However, there is a lack of tools that visualize both failure probability and order of minimal cut sets. Therefore, we presented in our previous paper “SafetyDomino” the design of a graphical metaphor for visualizing the minimal cut set’s order together with its failure probability. The design decisions were based on perception theory. To find the optimal color scheme for the design, we performed an empirical evaluation with 32 participants. This paper presents the extended version of an earlier paper [AZZD*12] with an extended analysis of the results and additional preference tests. The statistical results of this evaluation are extended to show the tendency IADIS International Journal on Computer Science and Information Systems 130 and relative accuracy of the participants using the different color schemes. These results indicate that the color scheme we call “gray-opposite” performs slightly better than the color scheme we call “black” for larger problem sizes. However, from the participants’ feedback and the additional preference tests (monitor-based and paper based) the black color was preferred. Therefore, we propose to interpret the results of any evaluation considering the feedback of the participants and including additional preference tests. In additional, we performed correlation tests to know if the demographic features of the participants had an effect to the results found.